Annotation models for crowdsourced ordinal data

In supervised learning when acquiring good quality labels is hard, practitioners resort to getting the data labeled by multiple noisy annotators. Various methods have been proposed to estimate the consensus labels for binary and categorical labels. A commonly used paradigm to annotate instances when the labels are inherently subjective is to use ordinal scales. In this paper we propose annotator models based on Receiver Operating Characteristic (ROC) curve analysis to consolidate the ordinal annotations from multiple annotators. The models lead to simple Expectation-Maximization (EM) algorithms that estimate both the consensus labels and annotator performance jointly. Experiments indicate that the proposed algorithm is superior to the commonly used majority voting rule.